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Abstract: The paper search for the minimum of the entropy of a two- 
dimensional distribution in the Frcchet class, the class of distributions with 
given marginals. The main result for discrete distributions is an algorithm 
for building the minimizing distribution, which is given by the maximum 
distribution function of the Frcchet class after a suitable rearrangement 
of the rows and of the columns. For absolutely continuous distributions a 
minimum does not exists, and the infimum is equal to — oo. 
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1. The problem 

Given an unidimensional discrete random variable X the entropy H{X) is de- 
fined by 

H{X) = -^Prlogpr 

r 

where the the Pr^s are the probabihties with which X takes its different values 
Xr- They form a countable set, with pr > 0, J2rPr — 1- Wc will usually employ 
for sums and integrals this simplified notation. If the value p — will occur, the 
product plogp will be taken by continuity to be zero. By its definition, entropy 
is non-negative; it may also be +oo. The values of the r.v. are not taken in 
account. The basis of the logarithm is inessential in this paper, since we proceed 
by comparison; for calculation we take e as basis. 

For discrete two-dimensional distributions the definition is: 

i?(x,y) = -^^p,,,iogp,,, 

r s 

As usual we pose 

Pr,- = P{X ^X,.) = ^Pr,s 
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p.^s = P{X = Xs) = y^Pr.s 



r 



Wc adopt also the notation H{P) were P ~ {pr,s] is the double array of 
the probabilities. Entropy is often defined as a function of a random variable, 
although it depends only, and partially, on its distribution. By its definition, a 
permutation of the values of the random variables does not change the value of 
the entropy. 

Entropy was defined by Claude E. Shannon [5] as a measure of uncertainty. 
This meaning is stressed by the fact that its minimum is zero (no uncertainty) 
when one of the p^s, is equal to 1, while its maximum, for a given number n 
of values, is attained when all the probabilities are equal; then the entropy is 
equal to \ogn. 

In statistics entropy is considered as a measure of uncertainty, more specifi- 
cally of dispersion. But its main utilization in the theory of information, where 
it plays a relevant role. In fact entropy changed of sign can indicate the amount 
of information given by the distribution. 

Information theory considers also the mutual information I{X, Y) defined by 



which shows that the search for the minimum of H{X,Y) furnishes also the 
maximum of I{X,Y). The maximum mutual information criterion is largely 
employed in applications. 

For the basic notions about entropy and information we refer to [6]. Remark 
that when we write minH or maxH without other indications we will mean 
always in the Frechet class, that is among the two-dimensional distributions 
with fixed marginals. 

We recall here some notions about distributions with given marginals [1]. If 
F(x) and G{y) are two one-dimensional distribution functions, we consider the 
set r(_F; G) of two-dimensional distributions functions which have F and G as 
marginals; T(F;G) = {F{x,y) : F{x,y) is a distribution function, 
lim^^oo F{x, y) = F{x); limx^oa F{x, y) = G{y)}. 

This is called also Frechet class, after a paper by Maurice Frechet [2] which 
started the study of the subject, after a relevant but almost ignored paper by 
Wassilly Hoeffding [3]. 

Following inequalities hold. 




Simple calculations bring to the relation 



I{X, Y) = H{X) + H{Y) - H{X; Y) 



max{F(,T) + F(y) -1,0} = W{x, y) < F{x, y) < M{x, y) = mm{F{x), G{y)} (1.1) 
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The functions on the right and on tlie left side belong to the class, being so in it 
the maximum and the minimum one. M{x,y) furnishes the maximum concentration, 
with a functional relation between X and Y: the distribution is concentrated on the 
set = G{Y)) , so that the relations Y = G-^F{X) and X = F~^G{Y) hold 

almost surely. Similar results hold for W . 

In the discrete case, to which we are interested now, the probabilities pr,s are subject 
to the inequalities 



Either bound can be reached for every couple (r, s), but not always at the same time 
for more than one couple. The distribution functions M and W correspond respectively 
to the cograduation table and the contrograduation table, introduced by T. Salvemini 
[4]. The cograduation table is constructed giving to pi,i its maximum value; then 
all the other items in the same row or in the same column, or both, are null, and 
the construction continues recursively. This construction is known also, specially in 
Operations Research, as NW rule. 

2. The dichotomic case 

We start with the simple case of a dichotomic table, i.e. both X and Y assume only 
two values. Then 

H{P) = logpi,l — Pl,2 logpi,2 — P2,l logP2,l - P2,2 logP2,2 

Expressing the probabilities as functions of the only variable we have 

H{P) = //(pij) = logpi,i - (pi,. — pi,i) log(pi,. - 

- (p.,! - pia) log(p.,i -pi,i) - (1 -pi,. -p,i +pi,i)log(l -pi,. -p.,1 +Pi,i) 

The derivative of H{pij) is 

-H''(pi,i) = - logpi.i - 1 + log(pi,. - pi,i) + 1 + log(p.,i - pi,i) + 1 - log(l - pi,. - p.,! +pi,i) - 1 
= log[(pi,. -pi,i)(p.,i -pi,i)] - log[pi,i(l -pi,. - p.,1 



max{pr,. +p.,s — 1,0} <Pr,s < min{pr,.,p.,s} 



(1.2) 



so that 



H'{pi,i) > (pi,. -pi,i)(p.,i -pi,i) -pi,i(l -pi,. - p.,1 +Pi,i) > pi,i < Pi,. p.,1 



It can be easily seen that pi,i = pi,.p.,i satisfies inequalities (1.2). So the maximum 
of H{P) is given by pi,i = pi,.p.,i, i.e. by the independence table. 
We must search now for the minimum. Suppose 



Pi,- > P2,., 



P-,1 > P-,2, 



Pi,- < 



(2.1) 



Then (1.2) becomes 



Pi,. +p-,i - 1 < Pi,i < Pi,- 
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and the minimum obtains for one of the extreme values. Denote respectively P' and 
P" the tables obtained putting pi^i = pi^. and pi,i = pi^. — 1. We may write 



H{P')-H{P") = -pi,. logpi,. - (p.,1 -pi,.)log(p.,i -pi,.) 

+ (pi,. +p.,i - l)log(pi,. +p.,i -!) + (! -pi,.)log(l -pi,.) 



and 

d 



[H{P')-H{P")] = -logpi, -l + log(p,i-pi,) + l + log(pi, +p.,i-l) 
dpi,. 

+ l-log(l-pi,)-l 

= log[(p.,i -Pi,.)(pi,. +p.,i - 1)] - log[pi,.(l - pi,.)] < 



(pi,. +p.,i - l)(p-.i -Pi.O - (1 -Pi,-)Pi,- =P^i < 

Then the maximum of H{P') — H(P") is given by the minimum of pi,. which, 
according to (2.1), is i, and 



H{P') - H{P") < ~ 




We have thus proved. 

Theorem 2.1. For a 2 x 2 distribution with given marginals the maximum of the 
entropy is given by the independence table. The minimum obtains choosing the maxi- 
mum pu,- among the pr,- 's and the maximum p.,„ among the p.,s 's and putting pu,v = 
min{p„,.p.,„}. 

The fact that the maximum of the entropy for discrete distributions is given by the 
independence distribution is well known, as the resulting inequality 

H{X, Y) < H(X) + H{Y) (2.2) 

and the intuitive meaning is immediate if we talk of information, since clearly the 
independence give the minimum information among all two-way distributions, and the 
minimum information cannot be lesser than the information already contained in the 
marginals. 

As for the minimum, for discrete distributions the following inequality hold 



proved by 



H{X,Y)>H{X),H{Y) 



(2.3) 
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H{X,Y)-H{X) = y^Pr,. logPr,. - y^Pr,a logPr,s 
r r,s 

= ^ Pr,- logPr,. - y~^Pr,alogPr 
r L s 

= ^ logPr," -logJJ^Prr/ 



3. Discrete distributions 



We consider now the case where X and Y assume a finite number of values, say 
{xi,X2, ■ ■ ■ ,Xm} and {1/1 , 2/2, • ■ • , J/n}- Some calculations along the same lines of the 
dichotomic case show that the maximum of the entropy is given by the independence 
distribution; a result well known, as already said. The search for the minimum by 
derivation is more complicate. Because of the form of the derivative, the maximum 
for each pr,s is reached in one of the extreme of (1.2), but the comparison is diffi- 
cult because it depends on the other values. We present an algorithm which gives a 
minimizing table. 

The hint is given by the 2x2 case; the algorithm consists of the repeated use of 
the same step. 



Algorithm 1 

Choose the maximum (or one of the maximums) among the Pr. 's, say pu.- 
Choose the maximum (or one of the maximums) among the p.,s's , say p. 
Put p*_j, = min{pu,-,p-,D} 

Delete the row, or the column, or both, in which there is only one entry different from 
zero and continue in the same way for subsequent tables. 



We remark that the table resulting after deleting is not a correlation table since 
the entries do not sum up to 1. We could obtain a correlation table by dividing all the 
entries by 1 ~Pu,- or 1 —p.,v\ but since we proceed by comparison it is easy to see that 
our construction arrives to the same result. 

If we rearrange the rows and the columns in the order in which we have taken the 
minimizing values P* is the cograduation table, built according the NW corner 
rule. It corresponds to the maximum distribution function in the Frechet class, i.e the 
function M in (1.1). 

Theorem 3.1. Given two r.v. ' X and Y with fixed discrete distributions, and H{X), 
H{Y) < +00, the minimum of the entropy H{X,Y) — H{P) is given by the correlation 
table P* built with the Algorithm 1. 

Proof. We start with the finite case, and we proceed by induction on the number of 
rows or columns; the theorem is trivially true when the number of rows, or of columns, 
is equal to 1. 
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Let us apply the algorithm, arriving to the correlation table P* . For sake of sim- 
plicity suppose that the first step brings to Pi^i = pi,., so that pi^s = for s > 1. In 
the table P* cancel the first column and call P* the remaining table. Also for a generic 
P call P the table obtained canceling the first column. We have 



HiP") = -pi,logpi,. +7y(P ) 
H{P) = -^pi,.logpi,, + (3.1) 



Now 



p^^' / \ PI'S 

Pi,.logpi,. - V'pi,^, logpi,^ = log =logTT ( — ) ^ 



so that 

-pi,. logpi,. < ^ -pi,s logpi,a (3.2) 

s 

Moreover, since P is built according to the Algorithm 1, by the recursive construc- 
tion H(P*) < H(P). This, along with (3.1) and (3.2), gives H{P*) < H{P), proving 
the theorem for the finite case. 

In the denumerable case, consider the set Z)„ ~ {{r, s) : r, s > n} and its comple- 
ment and write 



H{X,Y) = - y^Pr.a log Pr,s 

r,s 

= - ^ Pr,slogPr,s- ^ Pr,slogPr,s (3.3) 
{r,s)SD„ (r,s)eD^ 

From the hypothesis H{X),H{Y) < +00 and (2.2) it follows that H(X,Y) < +00, 
and this implies that the last sum in (3.3) tends to zero when n — >■ 00. Therefore 
applying the Algorithm 1 to the sum restricted to _D„ and going to the limit proves 
the theorem. 

□ 



4. Continuous distributions 

Entropy for continuous distributions is defined by 

H{X) = - ( f{x)\ogf{x)dx 

JR 

H{X,Y) = - / f {x,y) log fix, y)dxdy (4.1) 



«2 



The second definition in (4.1) contains the two-dimensional density function f{x, y), 
whose existence is not assured by the absolute continuity of the marginals, contrary 



G. Dall'Aglio and E. Bona/Minimum Entropy 



7 



to what happens for discrete distributions. This remark must be kept in mind if one 
proceeds by limit, as we wiU do: the limit of a sequence of absolutely continuous distri- 
butions is not necessarily absolutely continuous, even maintaining the same marginals; 
it can be, for instance, concentrated on a line, so that it has not a two-dimensional 
density function. 

Entropy for continuous distributions presents features very different from the dis- 
crete case. The differences arise from the fact that the density function may assume 
values greater than 1, so that the entropy may be negative. 

We will proceed by discretization and limit. Given a two-dimensional r.v. [X, Y) 
with density function f{x,y), which we will assume continuous, define 



r r r + 1 

X„ = - if -<X<^— r- = 0,±l,±2,... 

n n n 

Y„ ^ - if 1<X<^-^ s = 0,±l,±2,... (4.2) 
n n n 

Then Xn (resp. Yn) converges almost surely to X (resp Y) and (X„,y„) converges 
almost surely to {X,Y). For the probabilities, after defining 



L n 



r + 1 s s + 1' 
<x< , - <y < 



we have 



■(x„^Ly.^^-±1)^ [ 



f{x, y)dxdy 

f{x)dx (4.3) 

This makes clear the aforesaid difference between the discrete and the continuous 
case. The consequences of these remarks are illustrated by the following Theorem (see 
Theorem 1,3,1 of [6]). As we have said, now the entropy may be negative, so we require 
its boundness. 

Theorem 4.1. Given a r.v. (X.Y) with continuous density function f{x,y) and such 
that I H{X,Y) |< -l-oo, consider the sequence of r.v. 's {Xn,Y„} defined by (4-S) Then 
lim[H{X„,Y„) - 21ogn] = H{X,Y) 

Proof. We can write 

H{X„,Y„) -2logn = -y^Pn,r,slogPn,r,s - 2 log Tt 

r,s 

= — ^^pn,r,s l0g{n^Pn,r,s) (4.4) 

-('/. -H- 



Therefore we shall prove that 



G. Dall'Aglio and E. Bona/Minimum Entropy 



8 



lim I — / / / (2;, J/) log I v? / f{u,v)dudv \ dxdy 



.f{^,y)'^ogf{x,y)dxdy (4.5) 
Since f{x, y) is continuous, for (x, y) G A„,r,s we have 

f{u,v)dudv I = f{x,y) 



n— > + oo 



and 



lim f{x,y)log\n / /(u, = /(a;, ?/) log /(a;, j/) 



This is not sufficient for the convergence of the integral in (4.5). For proving it, we 
choose a positive integer m, and consider the set 



Dm = Ux,y) : — < f{x,y) < m\ 
I m J 

Since H{X, Y) is finite, we have 



lim / -f{x,y)logf{x,y)dxdy = H{X,Y) (4.6) 

For (it, v) belonging to Dm it is 

1 2 f 

< n / f{u, v)dudv < m 



m 



Then the integrand in (4.5) is bounded, and, by the dominated convergence theo- 
rem, for m fixed 



lim f- V / 



fix,y)log n / f{u,v)dudv dxdy ] = f(x,y) log f{x,y)dxdy 



This means that Ve > 0, m, 3n„ 



/ f{x,y)login'' / f{u,v)dudv\ dxdy - / f{x,y) log f{x,y)dxdy 

i.e. there is a subsequence {n„i} for which the limit (4.5) holds. But this can be said 
also if we start from a subsequence, and this prove the theorem. 

□ 



< e 
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We proceed now to the search for the mmimum of H[X,Y) when the marginal 
distributions F{x) and G{y) are given. We suppose that F and G are strictly increasing: 
if there are intervals {xr,Xr + dr) in which the density is zero, they are a countable 
set and are irrelevant for the entropy; they can be easily removed passing from F to a 
new distribution function defined by recurrence: 



Fo{x) = F{x) 
p (^\- f '^'(^) if a:: < x,; 

'^'^ > ^ \ F{x + dr) if X>X,. 

We proceed by discretization. According to Theorem 3.1, we have 

mini/(x„,F„) = //(x:,y;) 

where {X^, Y^) is built with the Algorithm 1. And because of Theorem 4.1 

mmH{X,„ y„) = lim [H{X:, y„*) - 21ogn] (4.7) 

n— >oo 

But the limit in (4.7) cannot be obtained through Theorem 3. As remarked for the 
Algorithm 1, the r.v. {Xn,Y^) has a distribution function which is maximum in the 
Frechet class, and the same is for the limit {X* , Y*) when n — s- oo . Therefore, as said 
then, (X*, Y*) is concentrated in the curve F = {F{X) — G{Y)), where F and G are 
the marginal distribution functions, and it has not a two-dimensional density, so that 
Theorem 4.1 cannot be applied. Since by hypothesis G is strictly increasing and has 
continuous derivative, the inverse G~^ exists, is strictly increasing, and has continuous 
derivative; the curve F can be written as 

Y = G'\F{X)) = t{X) 
where the function t has continuous derivative. 

The search for the minimum brings to a somewhat surprising result. 

Theorem 4.2. Let F{x) and G{y) be two one-dimensional distribution functions, 
with continuous derivatives, and such that H{X), H{Y) < +oo. Then the infimum of 
H{X, Y) in the Frechet class of the distribution functions with marginals F and G is 
—oo. More precisely there is a positive finite c such that 

mmH{X,Y) + logn^c (4.8) 
A minimum in the class does not exists, i. e. there is no distribution function with 
two-dimensional density which attains the minimum. 

Proof. The argument will be similar to the one in Theorem 4.1. For that we introduce 
the function 

.fn{x,y) = n'pn,r,a if {x , v) £ An,r,s 

SO that 



H{X*,Y*)-2logn = y^Pn,r,s log n^pl.r,, 



(4.9) 



E 



f*{x,y) log 



fniu, v)dudv 



dxdy 
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We prove first that out of the curve T the integral tend to 0: 





The argument trough the sets Dm as in Theorem 4.1 proves (4.10). 

We go on witli the proof, studying the structure of the distribution of the minimizing 
r.v. (X*,y^) . We may suppose that the rows and the columns are in the order in 
which they are chosen in the construction. This implies that cases with probability 
different from zero in the same row or in the same columns are adjacent. The first 
choice is now suppose that it is equal to p.,i; there may be other probabilities 

different from zero in the same column, so that p^^i^s ~ P s for s = 1, 2, . . . , fc (fc > 1); 
it is fc < oo since p5^_i,s — p-.s for all s would imply Pn.r.s ~ for r ^ 1 and every s, not 
consistent with the margins. After that another string will start. It may be vertical, 
as the first one, or horizontal, i.e. with p'^,r,B ~ Pr,-- We have therefore a set An of 
indexes r for which the strings are vertical, and for each s two values s^^ and s'^^r 
such that p^.r.s ~ P-,s , and a set Bn of indexes s for which the strings are horizontal, 
and for each s two values r'„^s and ,j such that Pn,r,s = Pr,- ■ One of the sets A„ and 
-Bn may be void. 

Suppose first that 






(4.11) 



This implies that the strings are disjoint, and 
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r,s 

= ^ P,;>log(np.,s) - ^ ^ Pr,.\og{npr,.) 



s + 1 



^ / ^(y) login/ g(«)dw j dt/ 

51 51 / " ( " / " J dx 

^ / 9{y)^ogg{y)dy 

fi^)^ogfix)dx 

,. + 1 r-;'_^ + i 

" 5Z 9 (y) ^og g {y) dy - ^ / ^ ^ f{x)logf{x)dx 

g{y)\ogg{y)dy~ j f(x)\og f{x)dx (4.12) 

where = IJ^^^^ (^i^, ^^^^^ and 5^ = IJ^^^^ . 

If we take a subsequence of {Ajj} converging to A and a subsequence of {B'„} 
converging to B , since H{X) and -ff(y) are finite the last term of (5.1) converge to 

- / 9{y)^ogg{y)dy- I f{x)log f{x)dx 

J A J B 

and this proves the tlieorem if (4.11) holds. Remark that (4.11) imply that P{A) + 
P{B) = 1. 

Suppose now that one of the equalities (4.11) do not hold, for instance 
This mean that p* or p* or both are greater than zero. But 

I I \n n J \ \n n J 



and 



p(A„,.,.nr) = p(t(9<y<^ 



This holds also for the other cases in the strings, both in An and in B„. Then all 
the strings are disjoint, and we are in the same situation as when (4.11) hold. And 
this concludes the proof. 

□ 
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5. Examples 

The complicate way to arrive to the minimum allows simple result only for special 
cases. 

Example 1. If X and Y have the same discrete distribution, then min H{X,Y) — 
H{X) = H{Y). 

Take the maximum, or one of the maximums, among the probabilities of each margin, 
say Then j = pi,. or p.,i. This cancels one row and one column, while the 

remaining marginal probabilities remain unchanged, so that we may continue in the 
same way. As a result, the minimizing distribution has p* = pr,- for any r, pp ^ = 
for r ^ s and 

H{X',Y*) = -^p,Vlogp,V = -X^P---' = = 

r 7' 

This expression shows two distributions, the first one two-dimensional, the other 
one unidimensional but the two entropies have the same value. 

Example 2. If X and Y have uniform discrete distributions, of length respectively m 
and kn , with k integer, then min H{X,Y) = H{X). 

We proceed by induction: if fc = 1, it is the case of Example 1, which now gives 
II(X) = logn. If A: > 1 , we have Pr ■ = — = -r- and p. , = - > —, therefore we can 
make pl i — Now the marginal probabilities are unchanged, except p.,i , which is 
equal to ^ — we can make p2,2 = and continue until Pn,n = The result is 
a square table of dimension n x n, plus a table of dimensions x n. This gives the 
result. 

If m = kn + r , with r different from zero, we may proceed as above for kn steps, and 
it remains a r x n table, with which we may proceed again in the same way. 

Example 3. We have found some cases in which II{X,Y) = H{X). Let us investi- 
gate when this happens. Suppose that X and Y have discrete distributions, of length 
respectively m and n. Then II(X, Y) = II{X) if there is a partition {/i, I2, ■ ■ ■ ■} such 
that 

^Pr,. =p.,s s = l,2, ... (5.1) 
If (5.1) is satisfied, we may put 

= Pr,. for r £ Is 

so that the distribution {Pr.s} is consistent with the marginals, and H{X* ,Y*) — 
II{X) gives the minimum by (2.3). On the other hand, if H{X, Y) — II{X), supposing, 
as already made, that the probabilities p.,s and pr,- are decreasing, it cannot be pi,. > 
p.,1 , since all the entries p*,^ would be lesser than pi,. and pi,. could not appear among 
the entries p*,s; continuing so we prove the assertion. We remark that, in agreement 
with (2.3), h\x) > H{Y). In fact 
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H(X)-H{Y) = ^p.,.logp.,, -^p,,.log 



E 



with equality liolding iff X and Y have the same distribution. 

A particular case is when X assumes two values and Y has geometric distribution, 
more exactly 

Pi,=p; P2,.=g = l-P 

P-,s = q^'^p; s = l,2, ... 

Now p.,1 = pi,. and p. ,2 — X/r>i P'"'' ^^^^^ Example 3 minH{X, Y) = H{X) 
= — logp — ^ log p. Also the direct calculation is very easy. 

Example 4. Consider now a case in which X and Y have respectively geometric and 
uniform distributions; more precisely 

1 /l^'-i 

Pr,- = — 



Then 



and, recursively, 



p.,s = -, s = l,2,...,fc 



. rfc- 1 11 1 



Pi,s=mm|— s = 2,3, . . . ,fc - 1 

The result is that the entries of P* are null, except 

Pi,s = P-,s for s = 1, 2, 3, . . . , fc - 1 

Pr,- = Pr,- for r = 2,3, ... 

Then 

H{X*,Y*) = - ^ p.,slogp.,s - y^Pr,. logpr,. 

l<s<fe-l r>2 

= - ^ P.,s logp.,s +P.,fc logp.,fc - y^Pr,. logPr,. +Pl,. logpi,. 
l<s<fe r>2 

= (X) + H{Y) + i log i + ^-i log ^— i = max //(X, F) - J^,. 
k k k k 
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with Kk > 0. Since Kk tends to zero when k oo, the minimum of H{X, Y) can be 
near to the maximum how much as we want. But the equality between the maximum 
and the minimum cannot be reached, because when fc — >■ oo one of the marginal 
distributions disappears. 
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